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DETAILED ACTION 
Response to Amendment 

This action is Final. Claims 1-9 are currently pending. Applicant's response 
received on 8/3/2007 is fully considered herein and is not persuasive. 



Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. (Original grounds) Claims 1-4 and 9 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Kotlikov et al. (US 2003/0012453 A1) in combination with 
Shimazu et al. (US005724454A). 



3. Applicant has amended independent claims 1 , 2 and 9 in each case inserting the 
single word "automatically" in specific steps of each method. 



1 . (Currently Amended) A method of detecting and using 
hanging wire pixels in a digital image, having pixels comprising: 

(a) automatically identifying pixels from the digital color image representing one or more 
sky regions; 

(b) automatically detecting pixels representing hanging wire regions in the sky regions; 
and 

(c) using the detected hanging wire pixels to determine the orientation of the digital image 
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or to replace such hanging wire pixels. 

2. (Currently Amended) A method of improving a digital 
color image having pixels, the method comprising: 

(a) automatically identifying pixels from the digital color image representing one or more 
sky regions; 

(b) automatically detecting sky occlusion regions by examining 
the sky regions; 

(c) developing a model based on the identified sky pixels, 

wherein such model is a mathematical function that has inputs of pixel position and 
outputs of color; and 

(d) using the model to operate on the digital color image to replace the values of pixels 
from the sky occlusion regions with values predicted by the model. 

9. (Currently amended) A method of removing hanging wire 

region pixels from detected sky regions in a digital color image having pixels, the method 
comprising: 

(a) automatically identifying pixels from the digital color image representing one or more 
sky regions; 

(b) automatically detecting hanging wire regions by examining 
the sky regions; 

(c) developing a model based on the identified sky pixels, 

wherein such model is a mathematical function that has inputs of pixel position and 
outputs of color; and 

(d) using the model to operate on the digital color image to replace the values of digital 
color image pixels associated with the hanging wire regions with values predicted by the 
model to thereby remove the hanging wire region pixels. 



In each instance, and for the claims depending (3 - 6), the addition of the word 

"automatically" does not actually add a limitation to the step, so the original rejection is 

maintained. There are two ways to define limits in a claim: 

For processes, the claim limitations will define steps or acts to be 
performed. For products, the claim limitations will define discrete physical 
structures or materials. (MPEP §2106 D C, 2 nd paragraph ) 
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Since the mere addition of the word "automatically" does not detail what steps 
are being introduced to each process to achieve this effect, the word carries no useful 
weight. The actions of an operator who "automatically" selects a region in an image in 
response to a program prompt would satisfy the requirement. 

Further, it is obvious and well within the abilities of one of ordinary skill in the art 
to automate manual steps in a program. If the word "automatically were to be given 
weight, the claims are still obvious under additional grounds mandated by the 
amendment. 

4. (New grounds) Claims 1-4 and 9 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kotlikov et al. (US 2003/0012453 A1) and Shimazu et al. 
(US005724454A) in combination with Ashton (US 2004/0066956). 

(New 103(a) rejection) Regarding amended claim 1, Kotlikov teaches [a] method 
of detecting thin occlusion pixels in a digital image, comprising: 

identifying pixels from the digital color image representing one or more 
background/non-object regions ("classification of image data into object and non-object 
regions", Kotlikov, paragraph 25, line 2); 

detecting pixels representing object regions in the background/non-object regions 
("non-object regions", Kotlikov, paragraph 25, line 3); and 
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using the detected non-object pixels to replace the object pixels ("amending the 
object data to more closely resemble the data of non-object regions", Kotlikov, 
paragraph 24, line 1 ). 

While Kotlikov describes the objects to be replaced as a "scratch or other defect 
in the image" at paragraph 005, Kotlikov does not expressly teach that his non-object 
regions are sky regions and his object regions are hanging wire regions. 

Shimazu, working the same field of endeavor of color image correction, does 
describe replacing hanging wire pixels in sky regions ("elimination of a undesirable 
image area in an image, for example, an electric wire running across the blue sky", 
Shimazu, column 26, line 2. 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to use the defect removal method of Kotlikov to solve the problem 
identified by Shimazu of removing the electric wire running across the blue sky in order 
to effect the "elimination of a undesirable image area in an image" (Shimazu, column 
26, line 2). 

The combination of Kotlikov and Shimazu does not explicitly teach 

(a) automatically identifying pixels and 

(b) automatically detecting pixels 

Ashton, working in a related problem solving area of region identification through 
image analysis does teach: 

(a) automatically identifying pixels and 
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(b) automatically detecting pixels 

("Step 106: Identification of an exemplar using manual tracing, semi-automated 
tracing, statistical region growth, or geometrically constrained region growth", Ashton 
paragraph 47, Ashton's system provides for either manually or automatically identifying 
a set of pixels as an object in an image. He teaches the concept of automating a 
manual process). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to apply the teaching of automating a manual operator selection 
step with the image correction system of the combination of Kotlikov and Shimazu in 
order to avoid the limitations of manual methods ("it is both tedious and time 
consuming", Ashton paragraph 4) and to avoid "results that are subject to both error and 
bias" (Ashton paragraph 4). 

Summary of Applicant's Remarks: Claim 1 has been modified to automatically 
identify sky region pixels and hanging wire regions. Kotlikov does not teach determining 
image orientation from hanging wires and sky regions. 

Shimazu teaches a contour extraction method for correcting color that would not be 
useful for detecting or removing hanging wires. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claim 1, and an alternate 103(a) rejection has 
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been presented above. As far as the combination of Kotlikov and Shimazu not teaching 
determining image orientation from handing wires and sky regions, this limitation in 
claim 1 is stated as an alternative, i.e. "c) using the detected hanging wire pixels to 
determine the orientation of the digital image or to replace such hanging wire pixels". 
The rejection as stated addresses the second stated option of replacing the hanging 
wire pixels. 

The Shimazu reference is used to explain an incentive for applying the system of 
Kotlikov to remove hidden wires. It would have been obvious at the time the invention 
was made to one of ordinary skill in the art to use the defect removal method of Kotlikov 
to solve the problem identified by Shimazu of removing the electric wire running across 
the blue sky in order to effect the "elimination of a undesirable image area in an image" 
(Shimazu, column 26, line 2). The functional elements of Shimazu are not actually part 
of the combination. 

The original rejection is maintained and an alternate 103(a) rejection is provided above. 

(New 103(a) rejection) Regarding claim 2, the Kotlikov-Shimazu and Ashton 
combination teaches the elements common with claim 1 as described above. It also 
teaches developing a model based on the identified sky pixels, wherein such model is a 
mathematical function that has inputs of pixel position and outputs of color ("the pixel 
may be replaced by the average or weighted average of pixels in its neighborhood", 
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Kotlikov, paragraph 50, line 3; and using the model to operate on the digital color image 
to replace the values of pixels from the sky occlusion regions with values predicted by 
the model ("the pixel may be replaced by the average or weighted average of pixels in 
its neighborhood", Kotlikov, paragraph 50, line 2). 

Summary of Applicant's Remarks: Claim 2 has been amended to automatically 
identify pixels in the sky region and pixels in occlusions. The combination of Kotlikov 
and Shimazu does not automatically identify sky regions and sky occlusions. The 
combination also does not replace sky occlusion regions. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claim 2, and an alternate 103(a) rejection has 
been presented above. Kotlikov does disclose replacing undesired pixels ("After they 
have been defined, the defect or object pixels may be corrected by any method known 
in the art. For example, the pixel may be replaced by the average or weighted average 
of pixels in its neighborhood, preferably excluding other defect pixels", Kotlikov, 
paragraph 50). In this context the pixels that make up the sky occlusion region are 
"defect" or "object" pixels and are replaced. As the pixels making up the region are 
replaced, then the region itself is replaced. 

The original rejection is maintained and an alternate 103(a) rejection is presented 
above. 
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(New 103(a) rejection) Regarding claim 3, the Kotlikov-Shimazu and Ashton 
combination teaches all the elements of claim 2 and further teaches wherein the model 
is a two-dimensional polynomial of the pixel position in the digital color image ("The 
defect may also be removed by interpolation such as with linear interpolation or 
quadratic interpolation", Kotlikov, paragraph 50, line 16). 

(New 103(a) rejection) Regarding claim 4, the Kotlikov-Shimazu and Ashton 
combination teaches [t]he method of claim 3 wherein the polynomial is a second-order 
polynomial ("The defect may also be removed by interpolation such as with linear 
interpolation or quadratic interpolation", Kotlikov, paragraph 50, line 16). 

Summary of Applicant's Remarks: Claims 3 and 4 depend from claim 2 has been 
amended to automatically identify pixels in the sky region and pixels in occlusions. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claim 2. 

The original rejections are maintained and alternate 103(a) rejections have been 
presented above. 



Application/Control Number: 10/816,317 Page 10 

Art Unit: 2624 

(New 103(a) rejection) Regarding claim 9, the Kotlikov-Shimazu and Ashton 
combination teaches [a] method of removing hanging wire region pixels from detected 
sky regions in a digital color image having pixels, the method comprising: 

All the elements that are common with claim 1 are described in the claim 1 
section above. It also teaches: developing a model based on the identified sky pixels, 
wherein such model is a mathematical function that has inputs of pixel position and 
outputs of color ("the pixel may be replaced by the average or weighted average of 
pixels in its neighborhood", Kotlikov, paragraph 50, line 3; and using the model to 
operate on the digital color image to replace the values of digital color image pixels 
associated with the hanging wire regions with values predicted by the model to thereby 
remove the hanging wire region pixels ("the pixel may be replaced by the average or 
weighted average of pixels in its neighborhood", Kotlikov, paragraph 50, line 3). 

Summary of Applicant's Remarks: Claim 9 has been amended to automatically 
identify pixels in the sky region and pixels in occlusions. The combination of Kotlikov 
and Shimazu does not automatically identify sky regions and sky occlusions. 
Reasoning similar to argument for claim 1. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claims 1 and 2 is discussed above and also 
applies to claim 9. 
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The original rejection is maintained. Additionally an alternate 103(a) rejection has been 
presented above. 

5. (Original grounds) Claim 5 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kotlikov et al. (US 20030012453 A1) and Shimazu et al. 
(US005724454A) in combination with Luo et al. (US 20030053686A1 ). 

6. (New grounds) Claim 5 is rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Kotlikov et al. (US 2003/0012453 A1). Shimazu et al. 
(US005724454A) and Ashton (US 2004/0066956) in combination with Luo et al. (US 
20030053686A1), 

(New 103(a) rejection) Regarding claim 5, the Kotlikov Shimazu and Ashton 
combination teaches all the elements that are in common with claims 1 and 2. 

The Kotlikov-Shimazu and Ashton combination does not teach (iii) using the 
model to operate on the digital color image to classify additional pixels not included in 
the initial sky region as sky. 

Luo, working in a similar problem solving area of color image segmentation, does 
teach using the model to operate on the digital color image to classify additional pixels 
not included in the initial sky region as sky ("an inventive region growing process is used 
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to fill in holes and extend boundaries of the candidate cloudy sky regions." Luo '686, 
paragraph 36). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to further combine the combination of Kotlikov, Shimazu and 
Ashton with the region growing technique of Luo'686 in order to deal with regions where 
" 'marginal' pixels may have sky-color and texture belief values that barely fail the global 
threshold but are close enough to the belief values of the neighboring pixels that have 
passed the initial global threshold", Luo'686, paragraph 36. 

Summary of Applicant's Remarks: Claim 5 depends on claim 2, and Lou initially 
classifies pixels as a sky region, then further refines this estimate by adding more pixels 
resulting in a larger region. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claim 2 is discussed above and also applies to 
claim 5. Applicant's observation that "Luo et al describes a model that initially classifies 
pixels of sky and then further adds to that initial sky region more pixels so that result is a 
larger sky region than what was originally classified" emphasizes the agreement with 
claim 5 which stated "(i) identifying pixels from the digital color image representing an 
initial sky region;" and "(iii) and using the model to operate on the digital color image to 
classify additional pixels not included in the initial sky region as sky." 
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The original rejection is maintained. Additionally an alternate 103(a) rejection has been 
presented above. 

7. (Original grounds) Claims 7 and 8 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Luo (US 200301 52289A1) and Miceli et al. (US 200301 281 49A1). 

Applicant has amended independent claim 7 inserting the single word 
"automatically" in specific steps of the method. 

7. (Currently amended) A method of determining the 

orientation of a digital color image having pixels, the method comprising: 

(a) automatically identifying pixels from the digital color image representing one or more 
sky regions; 

(b) automatically detecting one or more hanging wire regions by examining the sky 
regions; and 

(c) analyzing the hanging wire regions to determine the orientation of the digital color 
image. 

8. (New grounds) Claim 7 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over Luo (US 200301 52289A1) and Miceli et al. (US 200301 281 49A1) in 
combination with Ashton (US 2004/0066956). 
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(New 103(a) rejection) Regarding claim 7, Luo teaches [a] method of 
determining the orientation of a digital color image ("a method for determining the 
orientation of an image", Luo'289, paragraph 1) having pixels, the method comprising: 
identifying pixels from the digital color image representing one or more sky regions 
(Luo'289, Fig. 1, reference 221); detecting one or more non-sky regions by examining 
the sky regions (Luo'289, Fig. 7, "Other subject matters"); and analyzing the non-sky 
regions to determine the orientation of the digital color image (Luo'289, Fig. 7, 
Orientation from Semantic Object). 

Luo'289 does not expressly teach using hanging wire regions to determine image 
orientation, but he does teach that "image orientation can be determined from a plurality 
of semantic objects, including human faces, sky, text, sign, grass, snow field, open 
water, or any other semantic objects that appear frequently in images, have strong 
orientations by themselves, have orientations strongly correlated with image orientation, 
and last but not least, can be detected with reasonably high accuracy automatically." 
(Luo'289, paragraph 28). 

Miceli, working in a similar problem solving area of analysis of overhead line 
geometries, teaches that overhead lines do have a strong orientation ("The simplest 
features of the geometrical configurations are illustrated with FIG. 1(c), which shows an 
overhead line 22 suspended between two poles 10. The weight of the overhead line 22 
causes it to sag in a characteristic catenary shape, with the maximal sag from the points 
of suspension occurring midway between the poles 10." Miceli, paragraph 39). 
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It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to apply the method of Luo to make use of the features of power 
lines, in particular including hanging wires with their characteristic catenary sag as 
described by Miceli, as the basis for a semantic object detector as described by Luo 
("... semantic object detectors can be used to produce alternative or additional 
estimates of the image orientation." Luo, paragraph 25). 

Such a modification would have been obvious to one of ordinary skill in the art 
because power lines are a ubiquitous feature of everyday urban (and even rural) 
landscapes ("The power-utility-system infrastructure alone in North America includes 
approximately 150,000,000 wooden pole structures used to support overhead lines." 
Miceli, paragraph 3). Power lines also have a strong gravitational orientation as 
described by Miceli, just as the road "signs" and other features described by Luo. That 
is, Luo states, "Text and signs appear in many pictures, e.g., street scenes, shops, etc. 
In general, it is unlikely that signs and text are placed sideways or upside down ..." at 
paragraph 006. In like manner, power lines have a definite orientation with respect to 
gravity, and it is impossible that they would be hung "up-side-down", making them even 
more reliable than road signs. Such a strong gravitational orientation coupled with their 
prevalence renders a power line a reliable feature for determining orientation. One of 
ordinary skill and creativity in the art would clearly recognize this, even in the absence 
of the teaching of Miceli. 

The combination of Luo and Miceli does not explicitly teach 
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(a) automatically identifying pixels and 

(b) automatically detecting pixels 

Ashton, working in a related problem solving area of region identification through 
image analysis does teach: 

(a) automatically identifying pixels and 

(b) automatically detecting pixels 

("Step 106: Identification of an exemplar using manual tracing, semi-automated 
tracing, statistical region growth, or geometrically constrained region growth", Ashton 
paragraph 47, Ashton's system provides for either manually or automatically identifying 
a set of pixels as an object in an image. He teaches the concept of automating a 
manual process). 

It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to apply the teaching of automating a manual operator selection 
step with the image correction system of the combination of Luo and Miceli in order to 
avoid the limitations of manual methods ("it is both tedious and time consuming", Ashton 
paragraph 4) and to avoid "results that are subject to both error and bias" (Ashton 
paragraph 4). 

(New 103(a) rejection) Regarding claim 8, the combination of Luo, Miceli and Ashton 
teaches [t]he method of claim 7 wherein analyzing the hanging wire regions further 
comprises: determining the direction of gravity by examining the location of pixels of the 
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hanging wire region with respect to the endpoints of the hanging wire region ("The 
weight of the overhead line 22 causes it to sag in a characteristic catenary shape, with 
the maximal sag from the points of suspension occurring midway between the poles 
10." Miceli, paragraph 39). 

Summary of Applicant's Remarks: Claims 7 and 8 have been amended in a similar 
fashion to claim 1. Luo'289 is asserted to be unavailable under 35 USC 103(c). 
Applicant does not understand relevance of Miceli who uses radar signals. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claims 1 is discussed above and also applies to 
claim 7. 

(1.) Subject matter developed by another person, which qualifies as prior 
art only under one or more of subsections (e), (f), and fg) of section 102 of this title, 
shall not preclude patentability under this section where the subject matter and the 
claimed invention were, at the time the claimed invention was made, owned by the 
same person or subject to an obligation of assignment to the same person. (MPEP 
§103(c)(l)) 

Luo'289 does qualify as prior art under 35 USC 102(a) and is therefore not precluded by 
35 USC 103(c) and is an applicable reference. The teaching of Miceli is used merely to 
point out physical characteristics of hanging wires which one of ordinary skill in the art 
would be aware of. 
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The original rejections for claims 7 and 8 are maintained, and alternate 103(a) rejections 
have been presented above as well. 



9. (Original grounds) Claim 6 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of Kotlikov et al. (US 2003/0012453 A1 ) and 
Shimazu et al. (US005724454A) in combination with the combination of Luo (US 
200301 52289A1) and Miceli et al. (US 200301 281 49A1). 

10. (New 103(a) rejection) Claim 6 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of Kotlikov et al. (US 2003/0012453 A1 ), Shimazu et 
al. (US005724454A) and Ashton (US 2004/0066956) in combination with the 
combination of Luo (US 200301 52289A1) and Miceli et al. (US 200301 281 49A1). 

Regarding claim 6, the Kotlikov, Shimazu and Ashton combination teaches the 
elements of claim 2 as described above. The combination of Kotlikov, Shimazu and 
Ashton does not teach determining when sky occlusion regions are formed by hanging 
wires and determining the orientation of the image based on the detected hanging wire 
regions. 

The combination of Luo'289 and Miceli does teach determining when 
sky occlusion regions are formed by hanging wires and determining the orientation of 
the image based on the detected hanging wire regions (see rejection above for claim 7). 
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It would have been obvious at the time the invention was made for one of 
ordinary skill in the art to merge the hanging wire removal technique of the combination 
of the Kotlikov, Shimazu and Ashton with the image orientation method of the Luo'289 
and Miceli combination in order to avoid mis-oriented images since "it is aggravating if 
some of the images are displayed upside-down or sideways." (Luo'289, paragraph 3). 

Summary of Applicant's Remarks: Claims 6 depends from claim 2, which has been 
amended to automatically identify pixels in the sky region and pixels in occlusions. 

Examiner's Response: The problem with the lack of limitation introduced by the 
addition of the term "automatically" into claim 2. 

The original rejection is maintained and an alternate 103(a) rejection has been 
presented above. 



Conclusion 

1 1 . Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Thomas M. Redding whose telephone number is (571) 
270-1579. The examiner can normally be reached on Mon - Fri 7:30 am - 5:00 pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Brian P. Werner can be reached on (571) 272-7401. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/TMR/ 




